Imaging device and imaging method using compressed sensing

ABSTRACT

In an imaging device, a difference calculation unit calculates a differential signal between charge signals that have been accumulated and are held by first and charge holding units with different timings. A multiple sampling unit performs multiple sampling processing on the differential signal, and an analog digital conversion unit converts a signal that has undergone multiple sampling processing to a digital signal. That is, multiple sampling processing is performed on a differential signal with a higher sparsity than that of an image signal.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of International Application No.PCT/JP2014/002331 filed on Apr. 25, 2014, which claims priority toJapanese Patent Application No. 2013-112639 filed on May 29, 2013. Theentire disclosures of these applications are incorporated by referenceherein.

BACKGROUND

The present disclosure relates to an imaging device using compressedsensing.

In recent years, an image processing technique using “compressedsensing” has drawn attention. This technique is a technique in which aplurality of pixel values are added up and thus an image is captured,thereby compressing the image, and the image is reconstructed by imageprocessing. Normally, when additional imaging is performed, aninformation amount of an image is lost, the image quality of areconstruction image is greatly degraded. However, in compressedsensing, image reconstruction using the sparsity of the image isperformed, so that a reconstruction image with image quality notinferior to that of an uncompressed image may be obtained while theamount of data is reduced in additional imaging (see, for example, Y.Oike and A. E. Gamal, “A 256×256 CMOS Image Sensor with ΔΣ-BasedSingle-Shot Compressed Sensing,” IEEE International Solid-State CircuitsConference (ISSCC) Dig. of Tech. Papers, pp. 386-387, 2012).

The expression “an image is sparse” herein means a phenomenon in which,when an image is projected by a wavelet transform, a discrete cosinetransform (DCT), or the like, many coefficient values are substantiallyzero. As an image reconstruction method using the sparsity of an image,L0 norm minimization or L1 norm minimization is used in compressedsensing.

In compressed sensing, a data amount may be compressed by simpleadditional processing before an analog digital converter (which willabbreviated as “ADC”, as appropriate) in an image element, andtherefore, a drive frequency of ADC may be reduced. Thus, low powerconsumption, a high SN ratio, and reduction in communication band may berealized.

Japanese Unexamined Patent Publication No. 2010-245955 describes asolid-state image sensor using the concept of compressed sensing. In thesolid-state image sensor, a different wiring is coupled to each of aplurality of pixels, and a plurality of pixels of a pixel group aresequentially driven with timings with their phases being shifted andthus reads out a signal. With this configuration, a sample and holdcircuit is not needed, and degradation of image quality due to noiseincrease, increase in an area, and reduction in speed may be reduced.

A method in which compressed sensing is applied to an image usingImproved Iterative Curvelet Thresholding method is described in J. Ma,“Improved Iterative Curvelet Thresholding for Compressed Sensing andMeasurement,” IEEE Transactions on Instrumentation and Measurement, Vol.60, Iss. 1, pp. 126-136, 2011.

The following are related art documents: Japanese Unexamined PatentPublication No. 2010-245955; Japanese Unexamined Patent Publication No.2004-32517; Toshiyuki Tanaka, “Mathematics of Compressed Sensing,” IEICEFundamentals Review, vol. 4, no. 1, pp. 39-47, 2010; D. Takhar, J. N.Laska, M. B. Wakin. M. F. Durate, D. Baron, S. Sarvotham, K. F. Kelly,and R. G. Baraniuk, “A New Compressive Imaging Camera Architecture usingoptical-domain compression,” Proc. of Computational Imaging IV at SPIEElectronic Imaging, 2006; Y. Oike and A. E. Gamal, “A 256×256 CMOS ImageSensor with ΔΣ-Based Single-Shot Compressed Sensing,” IEEE InternationalSolid-State Circuits Conference (ISSCC) Dig. of Tech. Papers, pp.386-387, 2012; J. Ma, “Improved Iterative Curvelet Thresholding forCompressed Sensing and Measurement,” IEEE Transactions onInstrumentation and Measurement, Vol. 60, Iss. 1, pp. 126-136, 2011;Toshihide Ibaraki, Masao Fukushima, “Method of Optimization,”Information mathematics course, vol. 14, Kyoritsu Shuppan Co., Ltd., pp.159-164, Jul. 20, 1993, First Edition/First Copy; and MakotoNakashizuka, “Sparse Signal Representation and Its Image ProcessingApplication,” Journal of the Institute of Image Information andTelevision Engineers, Vol. 65, No. 10, pp. 1381-1386.

However, the sparsity of an image on which compressed sending ispremised is not necessarily achieved in a picture. For example, in animage with a high degree of randomness in which small objects scatter,the sparsity is poor. Therefore, in such an image, even when the methoddescribed in Toshiyuki Tanaka, “Mathematics of Compressed Sensing,”IEICE Fundamentals Review, vol. 4, no. 1, pp. 39-47, 2010, is used, aproblem arises in which the image quality of a reconstruction image isdegraded.

In order to solve the above-described problem, a technique disclosedherein has been devised, and it is therefore an object to increase theimage quality of a reconstruction image in an imaging device using acompressed sensing.

SUMMARY

According to an aspect, an imaging device includes, a photoelectricconversion unit configured to convert optical signals received by aplurality of pixels to electrical signals, a first charge holding unitconfigured to accumulate the electrical signals obtained by thephotoelectric conversion unit and hold the accumulated signals as chargesignals, a second charge holding unit configured to accumulate theelectric signals obtained by the photoelectric conversion unit with adifferent timing from accumulation by the first charge holding unit andhold the accumulated signals as charge signals, a difference calculationunit configured to calculate, for each of the plurality of pixels, adifferential value between the charge signal held by the first chargeholding unit and the charge signal held by the second charge holdingunit and obtain differential signals based on the differential values, amultiple sampling unit configured to perform, on the differentialsignals obtained by the difference calculation unit, multiple samplingprocessing which is processing for sampling signals of pixels eachlocated at a predetermined position from an original charge signal andadding up the sampled signals to generate a new signal, and an analogdigital conversion unit configured to convert an output signal of themultiple sampling unit to a digital signal.

Thus, in the imaging device, the difference calculation unit calculatesa differential signal between the charge signals held by the firstcharge holding unit and the second charge holding unit and obtaindifferential signals based on the differential values. The multiplesampling unit performs multiple sampling processing on the differentialsignals, the analog digital conversion unit converts a signal that hasundergone the multiple sampling processing to a digital signal. That is,multiple sampling processing is performed on a differential signal withhigher sparsity than that of an image signal. Therefore, imagereconstruction with high image quality may be realized using compressedsensing.

According to the present disclosure, in an imaging device usingcompressed sensing, a differential signal with a high sparsity is used,and therefore, a high quality reconstruction image may be obtained.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of an imagingdevice according to a first embodiment.

FIGS. 2A-2D are pattern diagrams illustrating an example of multiplesampling processing.

FIG. 3 is a pattern diagram illustrating pixel numbers.

FIGS. 4A-4D are pattern diagrams illustrating an example of multiplesampling processing.

FIGS. 5A-5D are pattern diagrams illustrating an example of multiplesampling processing.

FIG. 6 is a flow chart illustrating an example of processing performedin an imaging device according to the first embodiment.

FIGS. 7A-7C are pattern diagrams illustrating an example ofreconstruction processing using a difference image.

FIGS. 8A-8F are pattern diagrams illustrating atom in a curvelettransform in which information based on position, direction, and size ofan image is extracted.

FIGS. 9A-9C are pattern diagrams illustrating an example of an actualimage on which compressed sensing was performed, FIG. 9A illustrates anormal captured image, FIG. 9B illustrates an image obtained byreconstructing a picture after multiple sampling of was performedthereon, and FIG. 9C illustrates an image obtained by reconstructing adifference image after multiple sampling was performed thereon.

FIG. 10 is a block diagram illustrating a configuration of an imagingdevice according to a second embodiment.

DETAILED DESCRIPTION

According to a first aspect, an imaging device includes, a photoelectricconversion unit configured to convert optical signals received by aplurality of pixels to electrical signals, a first charge holding unitconfigured to accumulate the electrical signals obtained by thephotoelectric conversion unit and hold the accumulated signals as chargesignals, a second charge holding unit configured to accumulate theelectric signals obtained by the photoelectric conversion unit with adifferent timing from accumulation by the first charge holding unit andhold the accumulated signals as charge signals, a difference calculationunit configured to calculate, for each of the plurality of pixels, adifferential value between the charge signal held by the first chargeholding unit and the charge signal held by the second charge holdingunit and obtain differential signals based on the differential values, amultiple sampling unit configured to perform, on the differentialsignals obtained by the difference calculation unit, multiple samplingprocessing which is processing for sampling signals of pixels eachlocated at a predetermined position from an original charge signal andadding up the sampled signals to generate a new signal, and an analogdigital conversion unit configured to convert an output signal of themultiple sampling unit to a digital signal.

According to a second aspect, the imaging device according to the firstaspect further includes an image reconstruction unit configured toperform reconstruction processing on an output signal of the analogdigital conversion unit using multiple sampling information based onprocessing executed by the multiple sampling unit and obtain an imagesignal, and an output unit configured to output an image reconstructedby the image reconstruction unit.

According to a third aspect, in the imaging device according to thesecond aspect, the image reconstruction unit uses a projection transformfrom which information based on position, direction, and size of animage is extracted.

According to a fourth aspect, in the imaging device according to thethird aspect, the projection transform is a curvelet transform or aridgelet transform.

According to a fifth aspect, the imaging device according to the firstaspect further includes a processing switching unit configured to becapable of switching an input of the multiple sampling unit to, insteadof the differential signal, the charge signal held by the first orsecond charge holding unit.

According to a sixth aspect, in the imaging device according to thefifth aspect, at a start of imaging, the processing switching unit sets,instead of the differential signal, the charge signal held by the firstor second electric holding unit as an input of the multiple samplingunit.

According to a seventh aspect, in the imaging device according to thefifth aspect, the processing switching unit performs determination forthe switching using a signal value of the differential signal.

According to an eighth aspect, in the imaging device according to theseventh aspect, if the signal value of the differential signal isgreater than a predetermined value, the processing switching unit sets,instead of the differential signal, the charge signal held by the firstor second charge holding unit as an input of the multiple sampling unit.

According to a ninth aspect, an imaging method performed in an imagingdevice includes holding charge signals of a captured image withdifferent timings, calculating, for each pixel, a differential value forthe charge signals held with different timings and obtainingdifferential signals based on the different values.

Embodiments will be described below with reference to the accompanyingdrawings.

First Embodiment

FIG. 1 is a block diagram illustrating a configuration of an imagingdevice according to a first embodiment. The imaging device of FIG. 1includes a photoelectric conversion unit 101, a first charge holdingunit 102, a second charge holding unit 103, a difference calculationunit 104, a multiple sampling unit 105, an analog digital conversionunit 106, an image reconstruction unit 107, and an output unit 108.

The photoelectric conversion unit 101 includes a plurality of pixels,and each of the pixels converts a received optical signal to anelectrical signal. The plurality of pixels are realized, for example, byarranging photoelectric conversion elements, such as photo diodes, etc.,in a two-dimensional manner. With different timings, the first andsecond charge holding units 102 and 103 accumulate electrical signalsobtained by the photoelectric conversion unit 101 for a certain amountof time and hold the accumulated signals as charge signals. This may be,for example, realized by providing a plurality of memories used forholding charges and changes a memory that is to be used for each timingof imaging.

The difference calculation unit 104 calculates a differential valuebetween a charge signal held by the first charge holding unit 102 and acharge signal held by the second charge holding unit 103 for each pixeland obtains a differential signal based on the differential values. Thismay be realized by a known differential circuit. Note that an originalimage used for obtaining a difference has not been yet captured at astart of imaging, but in this case, an image at an end of previousimaging may be used, an image generated with a random number generatormay be given, or an image the entire screen of which is black and graymay be given.

The multiple sampling unit 105 performs multiple sampling processing ofthe differential signal obtained by the difference calculation unit 104and generates a new output signal. Multiple sampling informationindicating processing that is to be executed by the multiple samplingunit 105 is transmitted to the image reconstruction unit 107. The term“multiple sampling processing” herein means processing for samplingsignals of pixels each located at a predetermined position from anoriginal charge signal (the differential signal in this case) togenerate a new signal. The multiple sampling information includesinformation indicating a position of a pixel that has been sampled foruse in adding in an original charge signal for each of signal values ofnew output signals after multiple sampling processing has been performedthereon. Note that, when a gain is given at the time of adding, as willbe described later, the multiple sampling information may includeinformation for the given gain.

The multiple sampling unit 105 performs multiple sampling processing, sothat compression of image information may be performed and the amount ofsignal transmitted to the image reconstruction unit 107 may be reduced.The image reconstruction unit 107 uses received multiple samplinginformation, and thus, may reconstruct an image from compressed imageinformation.

FIGS. 2A-2D are pattern diagrams illustrating multiple samplingprocessing. For the sake of simplifying the description, processing with4×4 pixels, that is, 16 pixels, will be described as an example. FIGS.2A-2D illustrate readout pixels, that is, pixels used in multiplesampling processing, where t=1 to 4. Also, for the purpose ofillustration, pixel numbers are given to 4×4 pixels in FIG. 3. That is,in 4×4 pixels, “1,” “2,” “3,” and “4” are given in this order to thepixels arranged from the upper left corner in the direction toward theright, and similarly, “5,” “6,” . . . are given to the pixels arrangedfrom the left end in the next row, and “16” is given to the pixelarranged in the lower right corner.

In FIG. 2A, data of pixels with the pixel numbers 1, 6, 11, and 16 isread out and the data of the four pixels is subjected to addingprocessing, thereby generating an output signal of t=1. Similarly, inFIG. 2B, data of pixels with the pixel numbers 3, 8, 9, and 14 is readout and the data of the four pixels is added up, thereby generating anoutput signal of t=2. In FIG. 2C, data of pixels with the pixel numbers2, 5, 12, and 15 is read out and the data of the four pixels is addedup, thereby generating an output signal of t=3. In FIG. 2D, data ofpixels with the pixel numbers 4, 7, 10, and 13 is read out and the dataof the four pixels is added up, thereby generating an output signal oft=4.

In the above described manner, data of 4×4=16 pixels is compressed tofour pieces of data of t=1 to 4. Thus, the operation speed of the analogdigital conversion unit 105, which will be described later, may bereduced, so that an image with low noise may be reconstructed.

In the multiple sampling processing of FIGS. 2A-2D, multiple samplinginformation is, for example, as follows. As for t=1 to 4, given that apixel sampled for adding is denoted by “1” and a pixel which is notsampled is denoted by “0,” data coded in the order of the pixel numbersillustrated in FIG. 3 is the multiple sampling information. That is,

when t=1, “1000 0100 0010 0001,”

when t=2, “0010 0001 1000 0100,”

when t=3, “0100 1000 0001 0010,” and

when t=4, “0001 0010 0100 1000,”

and therefore, these are joined together to obtain the multiple samplinginformation, that is,

“1000 0100 0010 0001 0010 0001 1000 0100 0100 1000 0001 0010 0001 00100100 1000.”

Note that the format of multiple sampling information is not limited toone illustrated herein, but any format in which the position of a pixelsampled for use in adding is indicated may be used.

Note that in the example of FIGS. 2A-2D, each pixel is read out once inmultiple sampling processing. As a matter of course, in multiplesampling processing, the same pixel may be read out a plurality oftimes.

FIGS. 4A-4D are pattern diagrams illustrating an example of theabove-described multiple sampling processing. In FIG. 4A, data of pixelswith the pixel numbers 1, 3, 6, 8, 9, 11, 14, and 16 is read out and thedata of the eight pixels is added up, thereby generating an outputsignal of t=1. Similarly, in FIG. 4B, data of pixels with the pixelnumbers 2, 3, 5, 8, 9, 12, 14, and 15 is read out and the data of theeight pixels is added up, thereby generating an output signal of t=2. InFIG. 4C, data of pixels with the pixel numbers 2, 4, 5, 7, 10, 12, 13,and 15 is read out and the data of the eight pixels is added up, therebygenerating an output signal of t=3. In FIG. 4D, data of pixels with thepixel numbers 1, 4, 6, 7, 10, 11, 13, and 16 is read out and the data ofthe eight pixels is added up, thereby generating an output signal oft=4.

In the above described manner, the dynamic range of an output signal maybe increased by reading out each pixel a plurality of numbers of timesand performing adding processing, and therefore, noise may be reduced.Such multiple sampling processing is described, for example, in D.Takhar, J. N. Laska, M. B. Wakin. M. F. Durate, D. Baron, S. Sarvotham,K. F. Kelly, and R. G. Baraniuk, “A New Compressive Imaging CameraArchitecture using optical-domain compression,” Proc. of ComputationalImaging IV at SPIE Electronic Imaging, 2006, and Y. Oike and A. E.Gamal, “A 256×256 CMOS Image Sensor with ΔΣ-Based Single-Shot CompressedSensing,” IEEE International Solid-State Circuits Conference (ISSCC)Dig. of Tech. Papers, pp. 386-387, 2012.

In multiple sampling processing, a pixel position where sampling isperformed may be selected at random and/or independently. Thus,degradation of image information due to sampling processing may bereduced, and the image quality of a reconstruction image may be improved(see, for example, pp. 43-44 of Toshiyuki TANAKA, “Mathematics ofCompressed Sensing,” IEICE Fundamentals Review, vol. 4, no. 1, pp.39-47, 2010).

FIGS. 5A-5D are pattern diagrams illustrating an example of multiplesampling processing in which a pixel position where sampling isperformed is selected at random. In FIG. 5A, data of pixels with thepixel numbers 1, 4, 7, 8, 9, 10, 14 and 15 is read out and the data ofthe eight pixels is added up, thereby generating an output signal oft=1. Similarly, in FIG. 5B, data of pixels with the pixel numbers 2, 3,5, 8, 9, 11, 14 and 16 is read out and the data of the eight pixels isadded up, thereby generating an output signal of t=2. In FIG. 5C, dataof pixels with the pixel numbers 2, 3, 5, 6, 11, 12, 13 and 16 is readout and the data of the eight pixels is added up, thereby generating anoutput signal of t=3. In FIG. 5D, data of pixels with the pixel numbers1, 4, 6, 7, 10, 12, 13 and 15 is read out and the data of the eightpixels is added up, thereby generating an output signal of t=4.

Also, in multiple sampling processing, a gain may be given and weighingand adding may be performed, instead of merely adding up a plurality ofpieces of pixel data. When a plurality of pieces of pixel data are addedup, the dynamic range of data after adding has been performed increases,and the load of the analog digital conversion unit 106 increases. Inorder to solve this problem, it is effective to perform weighing andadding on pixel data. For example, when multiple sampling processingillustrated in FIGS. 2A-2D is performed, in order to cause the dynamicrange of data after adding has been performed to match that of originalpixel data, in normalization processing, a weight ¼ may be given as again.

The analog digital conversion unit 106 converts a signal generated inthe multiple sampling unit 105 to a digital signal. This processing maybe executed using a pipeline type or column type analog digitalconverter, which is widely known.

The image reconstruction unit 107 performs reconstruction processing onthe digital signal generated by the analog digital conversion unit 106using multiple sampling information transmitted from the multiplesampling unit 105, and obtains an image signal. Note that areconstruction image is a difference image, and therefore, for example,the reconstructed difference image is added to an image in a previousframe, thereby obtaining a final reconstruction image. Note that, forreconstruction processing in this case, a known technique, such as animproved iterative curvelet thresholding method (see, for example, J.Ma, “Improved Iterative Curvelet Thresholding for Compressed Sensing andMeasurement,” IEEE Transactions on Instrumentation and Measurement, Vol.60, Iss. 1, pp. 126-136, 2011), an affine scaling method (see, forexample, Toshihide Ibaraki, Masao Fukushima, “Method of Optimization,”Information mathematics course, vol. 14, Kyoritsu Shuppan Co., Ltd., pp.159-164, Jul. 20, 1993, First Edition/First Copy), etc., which arewidely used in compressed sensing, may be used.

The output unit 108 is an interface configured to display an imagereconstructed by the image reconstruction unit 107 on a display and tooutput the image for use in image processing, such as person detection,etc.

FIG. 6 is a flow chart illustrating an example of processing performedin an imaging device according to this embodiment. First, in Step S101,the photoelectric conversion unit 101 converts an optical signal to anelectric signal. Thus, a captured image is obtained. Then, if the firstcharge holding unit 102 is not used in a previous frame (NO in StepS102), the first charge holding unit 102 accumulates electrical signalsconverted by the photoelectric conversion unit 101 for a certain amountof time and holds the accumulated signals as charge signals (Step S103).If the first charge holding unit 102 is used in the previous frame (YESin Step S102), on the other hand, the second charge holding unit 103accumulates electrical signals converted by the photoelectric conversionunit 101 for a certain amount of time and holds the accumulated signalsas charge signals (Step S104).

That is, the first and second charge holding units 102 and 103 arecyclically used in a manner in which the second charge holding unit 103is used in a next frame in which the first charge holding unit 102 isused, and the first charge holding unit 102 is used in a next frame inwhich the second charge holding unit 103 is used.

Next, in Step S105, the difference calculation unit 104 calculates adifferential signal between a charge signal accumulated in the firstcharge holding unit 102 and a charge signal accumulated in the secondcharge holding unit 103. Thus, differential information for each pixelis calculated as a differential signal between frames that are adjacentto each other in terms of time.

Next, in Step S106, the multiple sampling unit 105 performs multiplesampling processing on the differential signal calculated by thedifference calculation unit 104 and generates a new output signal. Then,in Step S107, the analog digital conversion unit 106 converts a signalgenerated by the multiple sampling unit 105 to a digital signal.

Next, in Step S108, the image reconstruction unit 107 reconstructs animage from a digital signal generated by the analog digital conversionunit 106 using multiple sampling information transmitted from themultiple sampling unit 105. Then, in Step S109, the output unit 108outputs a reconstruction image to the outside of the imaging apparatus.

Advantages achieved when a difference image is used in compressedsensing will be described below.

It is known that, in compressed sensing, when an input image isprojected to a space, as sparsity of a coefficient vector thereofincreases, reconstruction image quality increases. That is, it isimportant in increasing reconstruction image quality to use a space inwhich an input image may be expressed in a sparse manner. The study ofthe present inventors has revealed that, in a difference image formed ofa differential signal of charge signals, sparsity is high.

FIGS. 7A-7C are pattern diagrams illustrating an example ofreconstruction processing using a difference image. FIG. 7A illustratesan image obtained by capturing a person P when t=1, and FIG. 7Billustrates an image obtained by capturing the person P when t=2. InFIG. 7C, a region A1 which is a difference image between the imageobtained when t=1 and the image obtained when t=2 and where there is adifferential value is illustrated in white.

In this case, if a subject is a person or an animal, a difference imageis expressed by a combination of curves, as in the region A1. Forexample, when viewing a block A2 of FIG. 7C, as a difference region,there is a circular arch which has about a half size of the size of theblock and the center of which is at right lower part at the right bottomof the block. Incidentally, a curve is expressed by three pieces ofinformation, that is, a position, an angle, and a size. On the basis ofthe foregoing, it is considered that, by projecting a difference imagein a space that may be expressed by position information, a directioncomponent, and size information, the difference image is caused to haveincreased sparsity. Therefore, as a space in which a difference image isprojected, a transform in which information based on the position,direction, and size of an image is extracted is used. Thus, sparsity ofthe image may be increased.

FIGS. 8A-8F are pattern diagrams illustrating some of atoms in curvelettransform, which is an example of conversion in which information basedon the position, direction, and size of an image is extracted. Notethat, in compressed sensing, an attention signal is expressed by alinear combination of a small number of vectors, and these vectors arecalled atoms. Note that, when these vectors are orthogonal, thesevectors are called basis vector. The difference image in the block A2 ofFIG. 7C looks similar to an atom illustrated in FIG. 8F, and thisindicates that curvelet transform is effective.

FIGS. 9A-9C are pattern diagrams illustrating an example of an actualimage on which compressed sensing was performed, FIG. 9A illustrates anormal captured image when multiple sampling was not performed, FIG. 9Billustrates an image obtained by reconstructing a picture using curvelettransform after multiple sampling was performed thereon, and FIG. 9Cillustrates an image obtained by adding an image in a previous frame onan image obtained by reconstructing a difference image using curvelettransform after multiple sampling was performed thereon.

In a picture illustrated in FIGS. 9A-9C, a global component is dominant,and therefore, the picture is not expressed in a sparse manner by alocal atom used for curvelet transform. For this reason, as illustratedin FIG. 9B, when a picture is used in compressed sensing, a large erroris generated in a reconstruction image. On the other hand, a differenceimage thereof is an image that may be expressed by local positioninformation, a direction component, and size information, and therefore,the difference image may be expressed in a sparse manner by curvelettransform. Accordingly, as illustrated in FIG. 9C, an image with highimage quality may be reconstructed.

Note that a projection transform used for image reconstruction is notlimited to a curvelet transform. In addition to a curvelet transform,for example, a ridgelet transform having a local atom may be used.

It is needless to say that, as a reconstruction technique, a knowntechnique, such as a matching pursuits method, a matching pursuitdenoising method, may be used (see, for example, Makoto Nnakashizuka,“Sparse Signal Representation and Its Image Processing Application,”Journal of the Institute of Image Information and Television Engineers,Vol. 65, No. 10, pp. 1381-1386).

As has been described above, according to this embodiment, using thefirst and second charge holding units 102 and 103 that accumulateelectrical signals obtained by the photoelectric conversion unit 101, adifferential signal of a captured image is obtained. Then, multiplesampling processing is performed on the differential signal to convert agenerated signal to a digital signal. By performing image reconstructionprocessing based on a compressed sensing technique on the digitalsignal, a reconstruction image with high image quality may be obtained.Furthermore, a space which has a local atom and in which informationbased on the position, direction, and size of an image is extracted isused as a projective space used in compressed sensing, so that areconstruction image with even higher image quality may be realized.

Note that, as an imaging device employing a plurality of charge holdingunits, a charge coupled device (CCD) is widely known. However, animaging device according to this embodiment calculates a differentialsignal of an image by using a plurality of charge holding units, andthus, increases sparsity of the image by executing multiple samplingprocessing on the differential signal. A totally different techniquefrom that for CCD is used in this embodiment.

A technique using differential information of a captured image using aplurality of memories is widely known. For example, Japanese UnexaminedPatent Publication No. 2004-32517 describes a technique in which, inorder to enable highly accurate focus control, a plurality of addedimages, average images, and difference images are used. However, this isa technique in which, after analog digital conversion is performed,difference processing or the like is performed on an obtained digitalsignal using a plurality of memories. On the other hand, an imagingdevice according to this embodiment performs difference processingbefore analog digital conversion is performed, and is totally differentfrom a known imaging device. That is, an imaging device according tothis embodiment uses a difference image in a compressed sensingtechnique, performs analog digital conversion after having performedmultiple sampling processing on a difference image, and restores animage with higher image quality from an obtained digital signal.

Second Embodiment

FIG. 10 is a block diagram illustrating a configuration of an imagingdevice according to a second embodiment. In FIG. 10, each componentelement which is similar to or the same as a corresponding component inFIG. 1 is denoted by the same reference character as that used in FIG.1, and the detailed description thereof will be omitted.

The imaging device of FIG. 10 further includes, in addition to thecomponent elements of the imaging device of FIG. 1, a processingswitching unit 109.

The processing switching unit 109 is capable of switching an input ofthe multiple sampling unit 105 to, instead of a differential signaloutput from the difference calculation unit 104, a charge signal held bythe first or second charge holding unit 102 or 103. That is, whether ornot multiple sampling processing is to be performed on a differentialsignal of respective charge signals held by the first and second chargeholding units 102 and 103, or whether multiple sampling processing is tobe performed on a charge signal held by the first charge holding unit102 or a charge signal held by the second charge holding unit 103 isswitched by the processing switching unit 109. When multiple samplingprocessing is performed on a charge signal held by the first or secondcharge holding unit 102 or 103, the difference calculation unit 104 maystop difference calculation.

The processing switching unit 109 performs determination of theswitching in accordance with a status of imaging. For example, at astart of imaging, the processing switching unit 109 sets a charge signalheld by the first or second charge holding unit 102 or 103 as an inputof the multiple sampling unit 105. Immediately after imaging is started,the image reconstruction unit 107 does not have an image of a previousframe, which is necessary for image reconstruction. Thus, at a start ofimaging, the processing switching unit 109 gives an output signal of thefirst or second charge holding unit 102 or 103 to the multiple samplingunit 105 and causes the multiple sampling unit 105 to performprocessing. Thus, a reconstruction image may be obtained even at a startof imaging, although image quality of the reconstruction image is low.Thereafter, by executing multiple sampling on a difference image, theimage quality of the reconstruction image may be increased.

The processing switching unit 109 may perform determination of theswitching using a signal value of a differential signal betweenrespective charge signals held by the first and second charge holdingunits 102 and 103. In this case, the processing switching unit 109 mayuse an output of the difference calculation unit 104.

For example, when the signal value of the differential signal is greaterthan a predetermined value, an output signal of the first or secondcharge holding unit 102 or 103, not the differential signal, is an inputof the multiple sampling unit 105. When the differential value is great,it is assumed that a subject has greatly changed due to a scene changeor an illuminant change. In this case, the differential signal is nolonger enough sparse, and therefore, it is more likely that, whenmultiple sampling processing is performed on the differential signal,image quality is rather degraded. Thus, an output signal of the first orsecond charge holding unit 102 or 103 is processed by the multiplesampling unit 105 without using the differential signal, therebymaintaining image quality of a reconstruction image.

On the other hand, when the differential value is small, a differentialsignal calculated by the difference calculation unit 104 is an input ofthe multiple sampling unit 105. When the differential value is small, asdescribed in the first embodiment, an image may be expressed in a sparsemanner by using a differential signal, and therefore, image quality of areconstruction image may be increased. Furthermore, the imaging devicemay be configured such that, when the differential value is close enoughto zero, multiple sampling processing for the image is not performed.This is because, in this case, an image of a previous frame may be usedas it is for reconstruction processing, and therefore, processing of theimage is not needed to be performed.

Note that, when processing switching using a signal value of adifferential signal is performed, a region unit used for determiningswitching may be, for example, a region in which multiple samplingprocessing is performed. For example, a total sum of signal values ofdifferential signals may be obtained for each region in which multiplesampling processing is performed, and determination for processingswitching in the region may be performed on the basis of the obtainedvalue. In this case, an image includes a region in which multiplesampling processing is performed on a differential signal and a regionin which multiple sampling processing is performed on an original chargesignal. Note that a region unit used for determining switching is notlimited to a region in which multiple sampling processing is performed,but a region in another predetermined range may be used as a unit. Asanother alternative, a total sum of signal values of differentialsignals may be obtained in an entire image, and determination forswitching processing for the entire image may be performed.

As has been described, according to this embodiment, the processingswitching unit 109 is provided, thereby allowing selection ofappropriate multiple sampling processing in accordance with a status ofimaging and change in input image, and therefore, image reconstructionwith higher image quality may be realized.

Note that, in each of the above-described embodiments, an imaging devicehas a configuration including an image reconstruction unit and an outputunit, but is not limited thereto. For example, a configuration that doesnot to include an image reconstruction unit may be employed such that adigital signal obtained by an analog digital conversion unit andmultiple sampling information are output to the outside of the imagingdevice.

Also, multiple sampling information used by the image reconstructionunit does not necessarily have to be information indicating processingitself that was executed by the multiple sampling unit. For example, theimage reconstruction unit may be configured to use multiple samplinginformation having a resolution that has been reduced to a level lowerthan that in processing executed by the multiple sampling unit. Thus, areconstruction image with a low resolution may be obtained. That is,multiple sampling information used by the image reconstruction unit maybe based on processing executed by the multiple sampling unit.

Note that, an imaging device described herein may not be realized as adevice. For example, the above-described operation of the imaging devicemay be performed by causing a general-purpose processor that is acomputer to execute a computer program recorded in a computer-readablerecording medium. The computer program includes, for example, an ordergroup that causes the computer to execute processing realized by theflow chart of FIG. 6. The computer program is recorded in a recordingmedium, such as a CD-ROM, etc., and is distributed as a product in themarket, or is transmitted via an electric communication line, such asthe Internet, etc.

The present disclosure is useful for enabling, in an imaging device,reconstruction of an image with high image quality while realizing lowpower consumption, a high SN ratio, and reduction in communication band.

What is claimed is:
 1. An imaging device, comprising: a photoelectricconversion unit including a plurality of pixels and configured toconvert optical signals received by the plurality of pixels toelectrical signals; a first charge holding unit configured to accumulatethe electrical signals, as first accumulated electrical signals,obtained by the photoelectric conversion unit and hold the firstaccumulated electrical signals as first charge signals; a second chargeholding unit configured to accumulate the electric signals, as secondaccumulated electrical signals, obtained by the photoelectric conversionunit with a different timing from accumulation by the first chargeholding unit and hold the second accumulated electrical signals assecond charge signals; a difference calculation unit configured toobtain differential signals based on differential values between thefirst charge signals and the second charge signals, by calculating, foreach of the plurality of pixels, a differential value between acorresponding first charge signal held by the first charge holding unitand a corresponding second charge signal held by the second chargeholding unit; a multiple sampling unit configured to perform, on thedifferential signals obtained by the difference calculation unit,multiple sampling processing which is processing for sampling signals ofpixels each located at a predetermined position from an original chargesignal and adding up the sampled signals to generate a new signal; ananalog digital conversion unit configured to convert an output signal ofthe multiple sampling unit to a digital signal; and a processingswitching unit configured to be capable of switching an input of themultiple sampling unit to, instead of the differential signal, thecharge signal held by the first or second charge holding unit, whereinat a start of imaging, the processing switching unit sets, instead ofthe differential signal, the charge signal held by the first or secondelectric holding unit as an input of the multiple sampling unit.
 2. Animaging device, comprising: a photoelectric conversion unit including aplurality of pixels and configured to convert optical signals receivedby the plurality of pixels to electrical signals; a first charge holdingunit configured to accumulate the electrical signals, as firstaccumulated electrical signals, obtained by the photoelectric conversionunit and hold the first accumulated electrical signals as first chargesignals; a second charge holding unit configured to accumulate theelectric signals, as second accumulated electrical signals, obtained bythe photoelectric conversion unit with a different timing fromaccumulation by the first charge holding unit and hold the secondaccumulated electrical signals as second charge signals; a differencecalculation unit configured to obtain differential signals based ondifferential values between the first charge signals and the secondcharge signals, by calculating, for each of the plurality of pixels, adifferential value between a corresponding first charge signal held bythe first charge holding unit and a corresponding second charge signalheld by the second charge holding unit; a multiple sampling unitconfigured to perform, on the differential signals obtained by thedifference calculation unit, multiple sampling processing which isprocessing for sampling signals of pixels each located at apredetermined position from an original charge signal and adding up thesampled signals to generate a new signal; an analog digital conversionunit configured to convert an output signal of the multiple samplingunit to a digital signal; and a processing switching unit configured tobe capable of switching an input of the multiple sampling unit to,instead of the differential signal, the charge signal held by the firstor second charge holding unit, wherein the processing switching unitperforms determination for the switching using a signal value of thedifferential signal.
 3. The imaging device of claim 2, wherein if thesignal value of the differential signal is greater than a predeterminedvalue, the processing switching unit sets, instead of the differentialsignal, the charge signal held by the first or second charge holdingunit as an input of the multiple sampling unit.
 4. An imaging device,comprising: a photoelectric conversion unit including a plurality ofpixels and configured to convert optical signals received by theplurality of pixels to electrical signals; a first charge holding unitconfigured to accumulate the electrical signals, as first accumulatedelectrical signals, obtained by the photoelectric conversion unit andhold the first accumulated electrical signals as first charge signals; asecond charge holding unit configured to accumulate the electricsignals, as second accumulated electrical signals, obtained by thephotoelectric conversion unit with a different timing from accumulationby the first charge holding unit and hold the second accumulatedelectrical signals as second charge signals; a difference calculationunit configured to obtain differential signals based on differentialvalues between the first charge signals and the second charge signals,by calculating, for each of the plurality of pixels, a differentialvalue between a corresponding first charge signal held by the firstcharge holding unit and a corresponding second charge signal held by thesecond charge holding unit; a multiple sampling unit configured toperform, on the differential signals obtained by the differencecalculation unit, multiple sampling processing which is processing forsampling signals of pixels each located at a predetermined position froman original charge signal and adding up the sampled signals to generatea new signal; and an analog digital conversion unit configured toconvert an output signal of the multiple sampling unit to a digitalsignal, wherein the multiple sampling unit selects the predeterminedposition at random in the multiple sampling processing.
 5. An imagingdevice, comprising: a photoelectric conversion unit including aplurality of pixels and configured to convert optical signals receivedby the plurality of pixels to electrical signals; a first charge holdingunit configured to accumulate the electrical signals, as firstaccumulated electrical signals, obtained by the photoelectric conversionunit and hold the first accumulated electrical signals as first chargesignals; a second charge holding unit configured to accumulate theelectric signals, as second accumulated electrical signals, obtained bythe photoelectric conversion unit with a different timing fromaccumulation by the first charge holding unit and hold the secondaccumulated electrical signals as second charge signals; a differencecalculation unit configured to obtain differential signals based ondifferential values between the first charge signals and the secondcharge signals, by calculating, for each of the plurality of pixels, adifferential value between a corresponding first charge signal held bythe first charge holding unit and a corresponding second charge signalheld by the second charge holding unit; a multiple sampling unitconfigured to perform, on the differential signals obtained by thedifference calculation unit, multiple sampling processing which isprocessing for sampling signals of pixels each located at apredetermined position from an original charge signal and adding up thesampled signals to generate a new signal; and an analog digitalconversion unit configured to convert an output signal of the multiplesampling unit to a digital signal, in the multiple sampling processing,the pixels each located at the predetermined position are include atleast one pixel which is not adjacent to other pixels and at least onepixel which is adjacent to at least one of other pixels.